Skip to content

[SPARK-59371][SQL] Gate the startsAndEndsWith LikeSimplification rewrite on a cheap child - #58663

Open
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-startsandendswith-cheap-gate
Open

[SPARK-59371][SQL] Gate the startsAndEndsWith LikeSimplification rewrite on a cheap child#58663
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-startsandendswith-cheap-gate

Conversation

@david-mollitor-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

LikeSimplification rewrites a LIKE 'prefix%suffix' pattern (the startsAndEndsWith
shape, e.g. 'a%b') into

lengthGuard(input) && StartsWith(input, prefix) && EndsWith(input, postfix)

which references the child input three times. The single-Like branch of
LikeSimplification.apply applied this rewrite unconditionally, unlike the
LikeAll/NotLikeAll/LikeAny/NotLikeAny branches, which only fire when
CollapseProject.isCheap(child) (SPARK-40228).

This PR gates the startsAndEndsWith case on CollapseProject.isCheap(input). For a
non-cheap child the pattern is left as a plain Like. The single-reference shapes
(startsWith, endsWith, contains, equalTo) reference the child once and remain
enabled for any child.

Why are the changes needed?

Duplicating a non-cheap child is both a correctness and a performance problem:

  • Correctness — a nondeterministic child (e.g. uuid(), cast(rand() as string)) is
    evaluated independently for each reference, so the three copies can produce different
    values and the rewritten predicate no longer matches the semantics of the original
    LIKE. Subexpression elimination does not help: it deliberately never deduplicates
    nondeterministic expressions.
  • Performance — an expensive deterministic child (e.g. sha2(col)) is written into
    the plan three times; the logical plan should not rely on subexpression elimination
    collapsing the repeats during codegen.

This is the same duplication class that SPARK-40228 fixed for the multi-LIKE rules.

Does this PR introduce any user-facing change?

Yes. For a LIKE 'prefix%suffix' pattern over a nondeterministic child, the child is now
evaluated once instead of once per reference. For example, uuid() LIKE 'a%b':

  • Before — rewritten to
    length(uuid()) >= 2 AND startswith(uuid(), 'a') AND endswith(uuid(), 'b'), drawing
    three independent UUIDs and testing the length, prefix, and suffix of different strings.
  • After — left as uuid() LIKE 'a%b', drawing a single UUID and matching it against
    the whole pattern.

The new behavior matches evaluating the LIKE directly (a single evaluation of the
child), which is what users expect. Queries with a nondeterministic argument to such a
LIKE can return different rows than before. Because the startsAndEndsWith rewrite is
long-standing, this is a user-facing change relative to released Spark versions as well as
master. Behavior is unchanged for cheap children (attributes, foldables, etc.), the
common case.

How was this patch tested?

Added two unit tests to LikeSimplificationSuite:

  • SPARK-59371: do not simplify startsAndEndsWith LIKE for a non-cheap child — asserts
    $"a".substring(1, 5) like "a%b" is left as Like (fails before this change).
  • SPARK-59371: still simplify single-reference LIKE shapes for a non-cheap child
    asserts the startsWith/endsWith/contains/equalTo shapes still simplify for a
    non-cheap child (guards against over-gating).

build/sbt 'catalyst/testOnly *LikeSimplificationSuite' passes (22 tests); scalastyle
clean.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

This pull request and its description were written by Isaac.

…ite on a cheap child

`LikeSimplification` rewrites `LIKE 'prefix%suffix'` (the `startsAndEndsWith` shape,
e.g. `'a%b'`) into `lengthGuard(input) && StartsWith(input, prefix) && EndsWith(input,
postfix)`, referencing the child three times. The single-`Like` branch of
`LikeSimplification.apply` applied this unconditionally, unlike the multi-LIKE branches,
which only fire when `CollapseProject.isCheap(child)` (SPARK-40228).

For a non-cheap child this duplicated the child: a nondeterministic child (e.g. `uuid()`)
was evaluated three times with different values -- changing results -- and an expensive
child (e.g. `sha2`) was evaluated repeatedly.

Gate the `startsAndEndsWith` case on `CollapseProject.isCheap(input)`, leaving the pattern
as a plain `Like` for a non-cheap child. The single-reference shapes evaluate the child
once, like the original `LIKE`, so they remain enabled for any child.

Generated-by: Claude Opus 4.8
Co-authored-by: Isaac <no-reply@databricks.com>
@david-mollitor-db

Copy link
Copy Markdown
Contributor Author

The failing CollationSQLRegexpSuite case here ("Like simplification should work with collated strings", the a%c rows) is not a flake — it's a direct consequence of this PR's change. The new startsAndEndsWith gate uses CollapseProject.isCheap(input), but isCheap does not recognize the Collate expression, so collate(col) LIKE 'a%c' is left as a plain Like instead of simplifying to And.

Collate is a zero-cost passthrough (its eval/codegen delegate to the child; the collation argument is never evaluated — it only re-tags the collation on the type), so it is genuinely as cheap to duplicate as the child it wraps.

I've opened #58712 (SPARK-59413) to fix that at the source: teach CollapseProject.isCheap to look through Collate to its value child.

#58712 is a prerequisite for this PR. Once it merges and this branch is rebased on master, the collated a%c rewrite returns and CollationSQLRegexpSuite passes with no change here — so this PR intentionally leaves that test unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant